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HOAD  TRACKING  AND  ANOMALY  DETECTION  IN  AERIAL  IMAGERY 


Lynn  H.  Quasi 
SRI  International 
Menlo  Park,  California 


ABSTRACT 

This  report  describes  a  new  procedure  for 
tracking  road  segments  and  finding  potential 
vehicles  in  imagery  of  approximately  1  to  3  feet 
per  pixel  ground  resolution.  This  work  is  part  of 
a  larger  effort  by  SRI  International  to  construct 
an  image  understanding  system  for  monitoring  roads 
in  aerial  imagery. 


INTRODUCTION 

This  report  describes  a  new  procedure  for 
tracking  road  segments  and  finding  potential 
vehicles  in  imagery  of  approximately  1  to  3  feet 
per  pixel  ground  resolution.  This  research  is  part 
of  a  larger  effort  by  SRI  International  to  build  a 
"knowledge  based  road  expert,"  described  by  Barrow 
and  Fischler  elsewhere  in  these  proceedings. 

The  overall  effort  is  directed  towards  specific 
problems  that  arise  in  processing  aerial 
photographs  for  such  military  applications  as 
cartography,  intelligence,  weapon  guidance,  and 
targeting.  A  key  concept  la  the  use  of  a 
generalised  digital  map  data  base  to  aid  in  the 
Interpretation  of  imagery. 


OBJECTIVES 

The  primary  objectives  of  the  overall  "road 
expert  system”  are  to  analyze  Images  to: 

(a)  Find  road  fragments  in  low-  to 
medium-resolution  iisages 

(b)  Track  roads  in  medium-  to  high- 
resolution  Images 

(c)  Find  anomalies  on  roads 

(d)  Interpret  anomalies  as  vehicles, 
shadows,  signposts,  surface 
markings,  etc. 

The  road  tracking  algorithm  discussed  here  is 
started  with  the  position  of  the  center  and 
direction  of  a  road  fragment  found  by  part  a).  The 
nominal  road  width  is  supplied  either  from  the  data 
base  or  by  an  image  analysis  function  that  can 
determine  the  width  of  a  road  fragment.  The  road 
tracker  produces  two  forms  of  output:  a  point  list 
describing  the  track  of  the  road  center  and  a 
binary  image  of  all  points  in  the  road  that  are 
anomalous  and  might  belong  to  vehicles.  In  the 


complete  road-expert  system,  this  image  will  then 
be  analyzed  by  part  d)  to  screen  false  alarms  and 
interpret  the  remaining  anomalies. 


ALGORITHM  DESCRIPTION 

Figure  la  shows  a  representative  road  scene 
containing  segments  of  a  multilane  freeway,  with  a 
few  vehicles  and  road  surface  markings  (painted 
arrows  and  words  in  the  leftmost  lane).  The  wear 
patterns  in  the  lanes  correspond  linearly  with  the 
road.  The  vehicles  and  other  anomalies  stand  out 
as  being  quite  different  from  the  pattern  of  the 
road. 

The  basic  road-tracking  algorithm  exploits  the 
above  observations.  Successive  road  intensity 
cross-sections  (RCS)  taken  perpendicular  to  the 
direction  of  the  road  showed  a  high  degree  of 
correlation,  which  suggested  that  road  tracking 
could  be  accomplished  by  using  cross-correlation. 
The  location  of  the  correlation  peak  was  used  to 
maintain  alignment  with  the  road  center  and  to 
generate  a  model  for  the  road  trajectory.  However, 
this  approach  turned  out  to  be  unsatisfactory 
because  small  alignment  errors  accumulated  and 
anomalies  perturbed  the  correlation  peak. 

To  overcome  these  problems,  four  refinements 
were  introduced : 

(a)  Cumulative  road  cross-section 
model 

(b)  Trajectory  extrapolation 

(o)  Anomaly  detection 

(d)  Masked  correlation. 

Instead  of  aligning  consecutive  RCSs,  each  RCS 
is  aligned  with  a  cumulative  RCS  model,  based  on  an 
exponentially  weighted  history  of  previously 
aligned  RCSs.  Parabolic  extrapolation  of  past 
correlation  peaks  is  used  to  predict  the  future 
road  trajectory.  The  predicted  trajectory  is  used 
to  guide  the  tracker  past  areas  where  the 
correlation  peak  is  unsatisfactory.  Anomalies  are 
detected  by  comparing  the  aligned  RCS  with  the  RCS 
model.  Corresponding  pixels  that  significantly 
disagree  are  marked  as  potential  anomalies.  The 
cross-correlation  is  then  repeated,  masking  out  the 
anomalous  pixels  to  obtain  a  more  accurate 
alignment. 

Steps  for  the  refined  tracking  algorithm  are 
given  below: 


(1)  Based  on  past  road  center  points 
and  directions,  extrapolate  the 
position  of  the  road  center  K  feet 
ahead. 

(2)  Extract  the  road  cross  section 
(RCS)  Intensities  along  a  line 
perpendicular  to  the  direction  of 
the  road  at  the  extrapolated 
center  point. 

(3)  Use  cross-correlation  to  find 
displacement  of  the  current  RCS 
with  respect  to  a  model  (RCS 
model)  that  Is  dynamically 
constructed  by  the  road  tracker. 

(4)  Generate  a  mask  Indicating  the 
positions  of  anomalous  pixels  that 
deviate  from  the  RCS  model. 

(5)  Recorr elate  over  the  unmasked 
pixels. 

(6)  Update  the  RCS  model  using  only 
the  valid  points  of  the  current 
RCS  at  the  best  alignment.  Update 
is  done  using  an  exponentially 
decaying  average. 

(7)  Adjust  the  position  of  the  road 
center  according  to  the  location 
of  the  correlation  peak. 

(8)  Detect  anomalies  as  being 
significant  deviations  from  the 
RCS  model. 

(9)  Repeat  steps  1-8  until  the  edge  of 
the  Image  Is  encountered  or  the 
RCS  model  becomes  Invalid. 


EXPERIMENTAL  RESULTS 

In  the  experiments  shown  here,  the  road  tracker 
was  interactively  started  by  indicating  the 
following  Information  for  each  road  segment: 

<XO,YO>  center  of  road  lane 
thetaO  direction  of  road  at  <X0,Y0> 

w  nominal  width  of  road 


The  freeway  example  in  Figure  1  conforms  well 
to  the  above  model,  as  shown  by  the  overlay  results 
in  Figure  1b.  The  bright  lines  Indicate  the  road 
trajectory  and  the  bright  blobs  Indicate  potential 
anomalies. 

The  simplistic  model  that  a  road  consists  of 
well-correlated  intensity  oross-seotions  clearly 
breaks  down  in  the  example  shown  in  Figure  2a, 
where  the  road  surface  changes  from  concrete  to 
asphalt  on  the  overpass.  Certainly  the  RCS  model 
generated  for  the  asphalt  will  not  match  the 
intensities  in  this  globally  changed  road  surface. 

When  the  tracker  encounters  the  surface  change  a 
high  percentage  of  the  pixels  in  the  RCS  will  be 


anomalous  (Figure  2b}.  When  this  occurs,  the 
tracker  extrapolates  ahead  and  tries  to  reacquire 
the  road.  If  the  road  is  not  reacquired  within  the 
length  of  the  longest  expected  anomaly,  the  tracker 
then  assumes  that  a  pavement  transition  has 
occurred  and  establishes  a  new  RCS  model. 

Host  of  the  anomalies  marked  in  Figure  2b  are 
due  to  road  surface  changes.  All  four  vehicles 
were  found  also.  A  later  section  will  discuss 
basic  changes  to  the  control  structure  of  the 
current  program  to  eliminate  the  false  alarms 
occurring  from  the  surface  changes. 

Figures  3a  and  3b  show  results  for  a  freeway 
interchange  on-ramp  loop.  This  example  is 
interesting  since  the  road  curves  rather  -tightly, 
and  the  road  surface  changes  at  approximately  the 
same  place  where  the  road  trajectory  changes  from  a 
circular  arc  to  a  straight  line. 

Figures  4a  and  4b  show  a  very  complicated 
example  of  road  forks,  changes  in  lane  width,  and 
intersections.  For  the  lanes  tracked,  all  vehicles 
and  at  least  portions  of  the  road  surface  marks 
(arrows  and  words)  were  found.  In  a  developed  road 
expert  system,  the  data  base  should  help 
significantly  in  handling  the  complexities  of  this 
image  by  knowing  the  locations  of  forks, 
intersections,  lane-width  changes,  etc.  This 
information  will  help  in  interpreting  the  cause  of 
RCS  model  changes. 

In  marked  contrast  with  the  situation  in  most  of 
the  previous  figures,  figure  5a  shows  a  rather 
poorly  defined  dirt  road  with  little  evidence  of 
wear  patterns.  Figure  5b  shows  the  successful 
results  of  the  road  tracker.  Host  of  the  anomalies 
marked  were  due  to  shadows  oast  by  sparsely 
foliated  trees. 


DISCUSSION 

The  preceding  examples  demonstrate  the 
capabilities  and  limitations  of  the  present 
tracking  algorithm.  The  algorithm  has  shown 
surprising  ability  to  contend  with  a  wide  variety 
of  road  situations,  including  total  change  in  the 
road  surface.  The  use  of  masked  eross-oorrelation 
techniques  eliminates  the  potential  perturbances  to 
the  road  track  by  anomalies.  Trajectory 
extrapolation  enables  the  tracker  to  reacquire  the 
road  after  detecting  that  the  road  surface  has 
changed.  All  results  were  obtained  using  the  same 
program  and  the  same  detection  and  threshold 
criteria;  no  attempt  was  made  to  "fine-tune"  the 
parameters  individually  for  each  example. 

One  defect  of  the  present  algorithm  is  the 
attempt  to  do  too  much  in  one  pass  along  the  road. 
In  particular,  in  the  present  system,  anomaly 
marking  begins  before  road-surfaoe  changes  have 
been  detected.  The  false  alarms  created  by  this 
defect  can  be  eliminated  either  by  backtracking 
when  a  road  transition  is  found,  or  by  performing 
the  detailed  anomaly  detection  as  a  second  pass 
along  the  road,  using  the  road-course  and  surface- 
change  knowledge  produced  by  the  tracker. 
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The  road  tracker  presently  operates  as  an 
independent  module.  As  a  component  of  a  larger 
road-expert  system,  it  will  be  started  from  the 
output  of  a  map-guided  road-detection  algorithm 
operating  on  lower-resolution  Imagery.  Data-base 
knowledge  can  also  be  used  in  the  tracking 
algorithm  to  Increase  reliability  and  reduce  false 
alarms  in  ancaaaly  detection.  Such  knowledge  might 
consist  of  previous  Imagery  of  the  same  area  or 
geometric  knowledge  about  locations  of  road  forks, 
intersections,  overpasses,  surface  changes,  lane- 
width  changes,  etc.  To  exploit  such  knowledge,  it 
is  necessary  to  establish  geometric  correspondence 
between  the  image  and  the  data  base  coordinate 
system.  If,  for  example,  a  road  anomaly 
corresponds  to  a  known  surface  marking  on  the  map 
or  appears  in  the  same  place  in  previous  images, 
then  it  is  probably  a  surface  marking  rather  than  a 
vehicle.  Similarly,  the  use  of  an  Illumination 
model  can  help  to  distinguish  objects  casting 
shadows  from  surface  markings. 

We  plan  to  acquire  and  digitize  images  taken 
under  diverse  viewing  conditions  such  as  partial 
cloud  coverage,  snow  cover,  oblique  viewing  angles, 
and  seasonal  variations.  This  will  introduce  a  new 
set  of  problems  for  the  tracking  algorithm  such  as 
non  visibility  of  road  segments  due  to  clouds  or 
occluding  objects  and  major  photometric  differences 
between  Images  of  the  same  area.  The  use  of  a  map 
data  base  and  sources  of  knowledge  will  be 
essential  to  guide  the  interpretation  of  such 
images. 

With  the  planned  enhancements  and  improvements, 
it  should  be  possible  to  detect  potential  vehicles 
with  very  high  hit  rates  and  low  false  alarm  rates 
in  difficult  imagery.  This  capability  is  a  central 
component  of  an  overall  road-monitoring  system. 
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